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ABSTRACT 



In response to concerns that Title I, the largest source of 
federal funding for education, was not significantly improving the 
educational achievement of at-risk students. Congress, in 1994, mandated 
major changes, including changes to the ways in which states measure student 
performance. This report provides a snapshot of how close states had come to 
meeting the 1994 requirements when the new requirements of the 2001 
legislation,, the No Child Left Behind Act, was signed into law. The General 
Accounting Office (GAO) has collaborated with other agencies in reviewing how 
states implement these measures and use the data to assess schools' progress 
in raising student achievement. This report contains GAO findings on the 
status of states' compliance with key 1994 Title I requirements, factors that 
have helped or hindered states in meeting requirements, and the actions 
states are taking to ensure that Title I assessments are scored accurately. 
Interviews with officials of the U.S. Department of Education and state Title 
I directors and other sources of data show that as of March 2002, there were 
17 states in compliance with 1994 Title I requirements, and 35 states and 
instrumentalities were not. Most of the states reported taking some action to 
ensure that Title I assessments were scored accurately, that any exemptions 
were justified, and that students with disabilities were receiving 
appropriate accommodations. Many states are still developing the procedures 
necessary to monitor such assessments. One appendix describes the scope and 
methodology of the study, and the other contains comments by the U.S. 
Department of Education to this report. (SLD) 
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United States General Accounting Office 
Washington, DC 20548 



April 1, 2002 

The Honorable Roderick R. Paige 
The Secretary of Education 

Dear Mr. Secretary: 

Title I of the Elementary and Secondary Education Act (ESEA), the largest 
source of federal funding for education, will provide states with $10.3 
billion in fiscal year 2002 to improve the educational achievement of 
children at risk. Title I serves about 12.5 million children in all 50 states, 
the District of Columbia, and Puerto Rico. In response to concerns that 
Title I funding was not significantly improving the educational 
achievement of at-risk students, in 1994 Congress mandated major 
changes to Title I, including changes to how states measure student 
performance. As part of the 1994 ESEA reauthorization, states were 
required to adopt or develop challenging curriculum content and 
performance standards, assessments aligned with content standards, and 
accountability systems to assess schools’ and districts’ progress in raising 
student achievement. In exchange for meeting these requirements, states 
were given increased flexibility in the use of Title I and other federal 
funds. New legislation that reauthorizes ESEA - The No Child Left Behind 
Act of 2001 - has since been passed. The new legislation does not reduce 
or eliminate any of the 1994 requirements. Instead, it augments the 
assessment and accountability requirements that states must implement 
and increases the stakes for schools that fail to make adequate progress. 
This report provides a snapshot of how close states had come to meeting 
the 1994 requirements when the 2001 requirements were signed into law. 

In light of the increasingly important role played by the required 
assessment and accountability measures, we have collaborated with other 
audit organizations to review aspects of how states are implementing 
these measures and ensuring that data used to assess schools’ progress in 
raising student achievement are complete and accurate. This report is one 
of several on this topic to be issued separately by the various collaborating 
audit organizations, which include the U.S. Department of Education’s 
Office of the Inspector General (OIG), the Texas State Auditor’s Office, the 
Pennsylvania State Auditor’s Office, and the City of Philadelphia 
Controller’s Office. In this report, we provide GAO’s findings regarding (1) 
the status of states’ compliance with key 1994 Title I requirements, (2) 
factors that have helped or hindered states in meeting the requirements, 
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and (3) the actions states are taking to ensure that Title I assessments are 
scored accurately, exemptions for students with limited English 
proficiency are justified and students with disabilities are appropriately 
accommodated during testing according to Title I regulations. 

To assess states’ progress in implementing important assessment and 
accountability requirements introduced by Title I, we (1) interviewed 
Education officials and reviewed Education memoranda and reports 
regarding states’ compliance 1 with the 1994 requirements, (2) surveyed 
State Education Agency (SEA) Title I directors regarding the challenges 
faced by states with and without approved assessment systems, and (3) 
conducted telephone interviews with Title I and other state officials in 
states that are in compliance with Title I to develop detailed information 
on strategies for overcoming key barriers. We also conducted similar 
interviews in some states still working to attain compliance with the 1994 
requirements to determine factors that have hindered their progress. We 
sent surveys to the 50 states, the District of Columbia, and Puerto Rico and 
obtained 50 responses (96 percent). We conducted our work in 
accordance with generally accepted government auditing standards 
between June 2001 and March 2002. (See app. I for specifics on our scope 
and methodology.) 



Results in Brief 



As of March 2002, 17 states were in compliance with the 1994 Title I 
assessment requirements; however, 35 states and instrumentalities were 
not. 2 The 1994 legislation required states to be in full compliance with the 
requirements by January 2001 but allowed the Department of Education to 
extend that deadline. For states that have not met the requirements, 
Education distinguishes between states that are near compliance and 
those that still have a significant amount of work remaining, granting 
timeline waivers to the former and compliance agreements to the latter. 
Education has granted timeline waivers to 30 states to give them more 
time to meet all requirements. According to Education, by January 31, 



1 In this report, we refer to a state as compliant when Education has fully approved its 
assessment system for meeting the 1994 final assessment requirements. We do not intend 
to imply that these states are necessarily in compliance with all requirements of the 1994 
law. 

2 In this report, when we refer to states we will be including the instrumentalities of the 
District of Columbia and Puerto Rico. 
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2004, all of these states will be in compliance and meet all 1994 Title I 
requirements, none of which are reduced or eliminated by the 2001 
legislation. If states fail to meet extended timelines for implementing the 
1994 requirements, they are subject to the withholding of some Title I 
administrative funds. Education has asked the remaining five states to 
enter into compliance agreements that will establish the final date by 
which they must meet all requirements. Education has held public 
hearings in each of these states. Under the law, without a compliance 
agreement, these states may lose funding. Noncompliant states most 
frequently have not met the specific requirements to assess all students 
and to report the data by subgroups of students, but some also have more 
intractable problems, such as assessments that are not aligned with state 
standards. Because the majority of states have not met the requirements of 
the 1994 law, many states may not be well positioned to meet the 
deadlines for implementing the additional requirements in the 2001 
legislation. 

Title I directors indicated that a state’s ability to meet the 1994 
requirements improved when the necessary players — both state leaders 
and state agency staff — made compliance a priority and coordinated with 
one another to achieve it; most directors indicated that state compliance 
was hindered by inadequate funding. Each compliant state that we 
interviewed said that the state’s governor, legislature, department of 
education leaders, or business leaders first prompted compliance 
initiatives, such as establishing a blue ribbon committee to address the 
issue. According to Title I directors, state leader initiatives that endorsed 
compliance as a high-priority, the backing of requirements by state and 
local staff involved in implementation, and technical expertise at the state 
level were among the types of support that contributed most to states’ 
compliance. Coordination between staff in different offices and levels of 
government was another factor identified as important to compliance. 
Inadequate funding was the most often cited factor that hindered state 
compliance with requirements, according to our survey of Title I directors. 
In interviews, the directors said that their investment of time and money in 
systems of assessment that predated and conflicted with the requirements 
of the 1994 legislation was an obstacle to compliance. 

Most of the states reported taking some action to ensure that Title I 
assessments were scored accurately, that any exemptions for students 
with limited English proficiency were justified, and students with 
disabilities were receiving appropriate testing accommodations. Almost all 
states hire a contractor to score the Title I assessments; however, 16 of 
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these states reported that they did not monitor the scoring done by the 
contractor. Most of those who did monitor the scoring reported they did 
so by selecting a sample of answer sheets to compare with the contractor’s 
results to ensure their accuracy. Others compared school and district test 
results with the results from previous years to identify large discrepancies 
that might suggest a problem. Several states have experienced errors in 
scoring done by contractors. For some states, these errors resulted in the 
incorrect identification of schools in need of improvement and students in 
need of additional services. Thirty-three states also reported taking some 
actions to ensure that any exemptions for students with limited English 
proficiency were justified and 41 reported actions to ensure students with 
disabilities received appropriate accommodations during testing. For 
example, states reported they had developed standards for districts to 
follow in accommodating these students so that assessments can yield 
valid measures of their performance. However, states reported few actions 
that would ensure that these guidelines were being followed. Many states 
are still developing procedures to ensure that any exemptions are justified 
and accommodations are appropriate. Education performs compliance 
reviews of grantee programs and is in the process of redesigning this 
review process. However, the redesigned reviews do not specifically 
include the monitoring of states’ actions with regard to contractors’ 
scoring of assessments. 

To reduce the potential for undetected errors in test scoring that could 
have material effects on educational decisions or damage confidence in 
the test results, we are recommending that Education specifically include 
monitoring of state actions regarding contractors and scoring provisions in 
its state compliance reviews. Education agreed to our recommendation in 
its official agency comments. A copy of the comments is printed in 
Appendix n. 



Background 



The original Title I legislation was passed in 1965, but the 1994 
reauthorization of ESEA mandated fundamental changes to the Title I 
program. One of the key changes involved the development of state 
systems of standards and assessments to ensure that students served by 
Title I were held to the same standards of achievement as all other 
children. Prior to 1994, some states had already implemented assessment 
systems, but these tended to be norm-referenced — students’ performance 




Page 4 



7 



GAO-02-393 Title I Assessments 



was judged in relation to the performance of other students. The 1994 
legislation required assessments that were criterion-based — students’ 
performance was to be judged against an objective standard. 3 Every state 
applying for Title I funds since 1994 agreed to implement the changes 
described in the 1994 law and to bring its assessment systems into 
compliance. States are also required to develop a definition of adequate 
yearly progress based on the assessments to hold schools accountable for 
educational progress. To help states that could not meet the proposed 
2001 timeline, Education had authority to grant timeline waivers and 
compliance agreements to states under certain conditions. In its 2001 
ESEA reauthorization, Congress increased testing requirements for states 
as well as the consequences for not improving test scores in schools and 
did not eliminate any of the requirements of the 1994 legislation. As shown 
in table 1, the 1994 and 2001 legislative requirements for assessment and 
accountability concern developing standards for content and performance; 
measuring improvement; implementing and administering assessments, 
including assessing students with limited English proficiency; reporting 
assessment data; and applying consequences for not meeting performance 
goals. 



3 A norm-referenced test is an objective test that is standardized on a group of individuals 
whose performance is evaluated in relation to the performance of others. Criterion- 
referenced tests are assessments that measure the mastery of specific skills or subject 
content and focus on the performance of an individual as measured against a standard or 
criterion rather than the performance of others taking the test. 
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Table 1 : Accountability and Assessment Requirements under the 1994 and 2001 Reauthorizations of Title 1 


1994 requirements 


2001 requirements 


Developing standards for content and performance 




Develop challenging standards for what students should know in 


In addition, develop standards for science content by 2005-06. 


math and reading or language arts. In addition, for each of these 


The same standards must be used for all children. 



standards, states should develop performance standards 
representing three levels: partially proficient, proficient, and 
advanced. The standards must be the same for all children. If the 
state does not have standards for all children, it must develop 
standards for Title I children that incorporate the same skills, 
knowledge, and performance expected of other children. 



Implementing and administering assessments 


Develop and implement assessments aligned with the content 
and performance standards in at least math and reading or 
language arts. 


Add assessments aligned with the content and performance 
standards in science by the 2007-08 school year. These science 
assessments must be administered at some time in each of the 
following grade ranges: from grades 3 through 5, 6 through 9, and 
10 through 12. 


Use the same assessment system to measure Title 1 students as 
the state uses to measure the performance of all other students. 
In the absence of a state system, a system that meets Title 1 
requirements must be developed for use in all Title 1 schools. 


Use the same assessment system to measure Title 1 students as 
the state uses to measure the performance of all other students. If 
the state provides evidence to the secretary that it lacks authority 
to adopt a statewide system, it may meet the Title 1 requirement 
by adopting an assessment system on a statewide basis and 
limiting its applicability to Title 1 students or by ensuring that the 
Title 1 local educational agency (LEA) adopts standards and 
aligned assessments. 


Include in the assessment system multiple measures of student 
performance, including measures that assess higher-order 
thinking skills and understanding. 


Unchanged. 


Administer assessments for math and reading in each of the 
following grade spans: from grades 3 through 5, 6 through 9, and 
10 through 12. 


Administer reading and math tests annually in grades 3 through 8, 
starting in the 2005-06 school year (in addition to the 
assessments previously required sometime within grades 10 



through 12). 

States do not have to administer math and reading or language 
arts tests annually in grades 3 through 8 if Congress does not 
provide specified amounts of funds to do so, but states have to 
continue to work on the development of the standards and 
assessments for those grades. 

Have students in grades 4 and 8 take the National Assessment 
for Educational Performance (NAEP) exams in reading and math 
every other year beginning in 2002-03, as long as the federal 
government pays for it. 



Implement controls to ensure the quality of the data collected from 
the assessments. 


Unchanged. 


Including students with limited English proficiency and with disabilities in assessments 


Assess students with disabilities and limited English proficiency 
according to standards for all other students. 

Provide reasonable adaptations and accommodations for 
students with disabilities or limited English proficiency, to include 
testing in the language and form most likely to yield accurate and 
reliable information on what they know and can do. 


By 2002-03, annually assess the language proficiency of students 
with limited English proficiency. Students who have attended a 
U.S. school for 3 consecutive years must be tested in English 
unless an individual assessment by the district shows testing in a 
native language will be more reliable. 
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1994 requirements 


2001 requirements 


Reporting data 


Report assessment results according to the following: by state, 
LEA, school, gender, major racial and ethnic groups, English 
proficiency, migrant status, disability, and economic disadvantage. 


Unchanged. 


LEAs must produce for each Title 1 school a performance profile 
with disaggregated results and must publicize and disseminate 
these to teachers, parents, students, and the community. LEAs 
must also provide individual student reports, including test scores 
and other information on the attainment of student performance 
standards. 


Provide annual information on the test performance of individual 
students and other indicators included in the state accountability 
system by 2002-03. Make this annual information available to 
parents and the public and include data on teacher qualifications. 
Compare high- and low-poverty schools with respect to the 
percentage of classes taught by teachers who are “highly 
qualified,” as defined in the law, and conduct similar analyses for 
subgroups listed in previous law. 


Measuring improvement 


Use performance standards to establish a benchmark for 
improvement referred to as “adequate yearly progress.” All LEAs 
and schools must meet the state’s adequate yearly progress 
standard, for example, having 90 percent of their students 
performing at the proficient level in math. LEAs and schools must 
show continuous progress toward meeting the adequate yearly 
progress standard. The state defines the level of progress a 
school or LEA must show. Schools that do not make the required 
advancement toward the adequate yearly progress standard can 
face consequences, such as the replacement of the existing staff. 


In addition to showing gains in the academic achievement of the 
overall school population, schools and districts must show that the 
following subcategories of students have made gains in their 
academic achievement: pupils who are economically 
disadvantaged, have limited English proficiency, are disabled, or 
belong to a major racial and ethnic group. To demonstrate gains 
among these subcategories of students, school districts measure 
their progress against the state’s definition of adequate yearly 
progress. 

States have 12 years for all students to perform at the proficient 
level. 


Consequences for not meeting the adequate yearly progress standard 


LEAs are required to identify for improvement any schools that fail 
to make adequate yearly progress for 2 consecutive years and to 
provide technical assistance to help failing schools develop and 
implement required improvement plans. After a school has failed 
to meet the adequate yearly progress standard for 3 consecutive 
years, LEAs must take corrective action to improve the school. 


New requirements are more specific as to what actions an LEA 
must take to improve failing schools. Actions are defined for each 
year the school continues to fail leading up to the 5th year of 
failure when a school must be restructured by changing to a 
charter school, replacing school staff, or state takeover of the 
school administration. The new law also provides that LEAs offer 
options to children in failing schools. Depending on the number of 
years a school has been designated for improvement, these 
options may include going to another public school with 
transportation paid by the LEA or using Title 1 funds to pay for a 
private tutor. 



Source: P.L. 103-382 and P.L. 107-110. 



Almost all states employ contractors to perform services to help them 
meet these requirements. Among states that we interviewed, contractors 
included private companies, universities, nonprofit organizations, and 
individual consultants. These entities were hired to provide services that 
may include assessment development, administration, scoring, analysis, 
and reporting of results. Some of these entities can provide combinations 
of services to states, such as test development and test scoring. States are 
responsible for monitoring contractor performance. 
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Congress allowed states to phase in the 1994 ESEA requirements over 
time, giving states until the beginning of the 2000-01 school year to fully 
implement them with the possibility of limited extensions. Education is 
responsible for determining whether or not a state is in compliance with 
these requirements and is authorized under ESEA, to give states more time 
to implement the requirements as long as states are making adequate 
progress toward this goal. States submit evidence to Education showing 
that their system for assessing students and holding schools accountable 
meets Title I requirements. Education has contracted with individuals with 
expertise in assessments and Title I to review this evidence. The experts 
provide Education with a report on the status of each state regarding the 
degree to which a state’s system for assessing students meets the 
requirements and deserves approval. Using this and other information, the 
Secretary sends each state a decision letter that summarizes the experts’ 
review and communicates whether a state is in full compliance, in need of 
a timeline waiver, or more seriously, a compliance agreement. Education 
may withhold funds if a state does not meet the terms of its compliance 
agreement. The 1994 legislation was not specific in the amount of 
administrative funds that could be withheld from states failing to meet 
negotiated timelines, but the 2001 legislation states that Education must 
withhold 25 percent of state administrative funds until the state meets the 
1994 requirements including the terms of any timeline waivers or 
compliance agreements. 

In June 2000, we issued a report on states’ efforts to ensure compliance 
with key Title I requirements. 4 At that time, we expressed concern about 
the number of states that were not positioned to meet the deadlines in the 
1994 law. To increase compliance, we made two recommendations. We 
recommended that the Department of Education should (1) facilitate 
among states the exchange of information and best practices regarding the 
implementation of Title I requirements and (2) implement additional 
measures to improve research on the effectiveness of different services 
provided through Title I to improve student outcomes. Education 
continues to work on the implementation of these recommendations. In 
addition, we said that Congress should consider requiring that states’ 
definitions of adequate yearly progress apply to disadvantaged children, as 



4 General Accounting Office, “Title I Program: Stronger Accountability Needed for 
Performance of Disadvantaged Students” GAO/HEHS-OO-89, (Washington, D.C.: June 1, 
2000 ). 
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well as to the overall student population. The 2001 legislation does require 
that states apply adequate yearly progress requirements and report on the 
results by subgroups, including students in poverty, with disabilities, and 
with limited English proficiency. 



As of March 2002, 17 states were in compliance with the 1994 Title I 
assessment requirements; however, 35 were not. (See table 2.) 
Departmental approval of timeline waivers to give states more time to 
reach compliance has been granted for 30 states. Education has asked five 
states to enter into compliance agreements that will establish the final 
date by which they must be in compliance before losing Title I funding. 
Among other requirements, states that are not in compliance have most 
frequently not met the specific requirements to assess all students and 
break out assessment data by subcategories of students. The 2001 
legislation requires states to implement additional assessments through 
2008, thus substantially augmenting current assessment requirements. 
Education has published a notice of proposed negotiated rulemaking in 
the Federal Register and has solicited comments from outside parties in 
preparation for establishing state compliance standards for the 2001 
legislation. 
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Table 2: Status of States’ Compliance with 1994 Title 1 Assessment Requirements 
as of March 2002 


Compliant (17) 


Noncompliant (35) 


Colorado 


Alabama 


Nebraska 


Delaware 


Alaska 


Nevada 


Indiana 


Arizona 


New Hampshire 


Kansas 


Arkansas 


New Jersey 


Louisiana 


California 


New Mexico 


Maine 


Connecticut 


New York 


Maryland 


District of Columbia 


North Dakota 


Massachusetts 


Florida 


Ohio 


Missouri 


Georgia 


Oklahoma 


North Carolina 


Hawaii 


Puerto Rico 


Oregon 


Idaho 


South Carolina 


Pennsylvania 


Illinois 


South Dakota 


Rhode Island 


Iowa 


Tennessee 


Texas 


Kentucky 


Utah 


Vermont 


Michigan 


Washington 


Virginia 


Minnesota 


West Virginia 


Wyoming 


Mississippi 


Wisconsin 


Montana 



Source: Department of Education. 



When Education determines that a state is not in compliance with the 1994 
Title I assessment requirements, it may grant the state a timeline waiver 
for meeting those requirements. A waiver may not exceed 3 years. 
Education officials indicate that the agency grants waivers to states that 
have a history of success in implementing significant portions of their 
assessment systems, have a clear plan with a definite timeline for 
complying with the Title I requirements, and have the capacity to carry out 
the plan and thus meet those requirements. When a state requests a 
waiver, it must provide Education with a plan that includes a timeline for 
addressing deficiencies in the state’s assessment system. Education 
reviews this information to decide whether the waiver should be granted 
and its duration. So far, Education has granted timeline waivers to 30 
states, (see table 3.) 
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Table 3: Timeline Waiver Status for 30 States in Meeting the 1994 Title 1 Assessment 
Requirements as of March 2002 


1-year (9) 


2-year (18) 


3-year (3) 


Arkansas 


Alaska 


Nevada 


Minnesota 


Connecticut 


Arizona 


New Jersey 


Ohio 


Florida 


California 


New Mexico 


Puerto Rico 


Illinois 


Georgia 


North Dakota 




Kentucky 


Hawaii 


Oklahoma 




New Hampshire 


Iowa 


South Carolina 




New York 


Michigan 


South Dakota 




Utah 


Mississippi 


Tennessee 




Washington 


Nebraska 


Wisconsin 





Note: Each timeline waiver has an exact ending date. 

1 year = to be complete December 31 , 2002 or sooner. 

2 year = to be complete December 12, 2003 or sooner. 

3 year = to be complete January 31 , 2004 or sooner. 

Source: Department of Education. 

A compliance agreement is deemed necessary when Education determines 
that a state will not complete the implementation of its assessment system 
in a timely manner. According to Education officials, a state requiring a 
compliance agreement generally does not have a history of successful 
implementation, has not met a significant number of Title I requirements, 
and does not have a plan in place for meeting those requirements. 
Education recommends a compliance agreement so that a state may 
continue to receive Title I funds. Before Education may enter into a 
compliance agreement, a public hearing must be held in which the state 
has the burden of persuading Education that full compliance with the Title 
I requirements is not feasible until a future date. The state must be able to 
attain compliance within 3 years of the signing of the compliance 
agreement by the state and Education. The state then negotiates the terms 
of the agreement with Education. Education’s written findings and the 
terms of the compliance agreement are published in the Federal Register. 
A state that enters into a compliance agreement to address requirements 
of the 1994 Title 1 law and subsequently fails to meet the requirements of 
the agreement can be subject to loss of some state Title I administrative 
funds. Education is presently working on five compliance agreements 
(Alabama, Idaho, Montana, West Virginia, and the District of Columbia) 
and has held public hearings for each. 



Page 11 



14 



GAO-02 -3 93 Title I Assessments 



The 2001 reauthorization of ESEA was signed into law on January 8, 2002. 
The act provides states not in compliance with the 1994 Title I 
requirements at the time of the signing of the 2001 legislation with a 90-day 
period that started on January 8, 2002 to negotiate changes in the dates by 
which they must be in compliance with the 1994 requirements. After the 
conclusion of this 90-day period, the legislation prohibits further 
extensions for compliance with the 1994 requirements. States failing to 
meet these negotiated timelines will be subject to loss of some of their 
Title I administrative funds. According to senior Education officials, this 
loss could be significant to states, as many use federal program 
administrative funds to pay the salaries of state department of education 
staff. 

A review of documents from Education shows that noncompliant states 
have most commonly not met two Title I requirements — assessing all 
students and breaking out assessment data by subcategories of students. 
Title I does not permit states to exempt any student subgroup from their 
assessments and Education’s guidance states that individual exemptions 
may be permitted by the states in extraordinary circumstances. 
Nonetheless, many states allow substantial exemptions for students with 
disabilities and limited English proficiency. Several states reported that 
they have only recently amended laws that prohibited testing of some 
students with limited English proficiency. Title I also requires states, local 
districts, and schools to report the performance of students overall and in 
a variety of subcategories. These categories are gender, race, ethnicity, 
English proficiency status, migrant status, disability status, and economic 
disadvantage. Many states disaggregated data for some but not all of these 
categories. Documents from Education show that data for the disabled, 
migrant, and economically disadvantaged subcategories are the most 
common subgroups excluded from state, district, and school reports. In 
addition, many states lag in other areas, such as aligning assessments to 
state content standards. 

To achieve compliance with the 2001 legislation, states will need to add 
new standards and increase assessment efforts, as detailed in table 1. In 
responding to our survey, 48 states reported that they have developed 
content standards in science, but only 16 reported having annual 
assessments for math and 18 reported annual assessments for reading in 
all grades 3 through 8. In addition, states will not have the 2 to 3 year 
timeline waivers available to them as they had when they worked to meet 
the 1994 requirements. New 2001 requirements listed in table 1 have 
deadlines that vary according to the requirement, and the Secretary of 
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Education can give states 1 additional year from those deadlines to meet 
the new requirements, but only in case of a “natural disaster or a 
precipitous and unforeseen decline in the financial resources of the state.” 
Since the majority of states have not met the requirements of the 1994 law, 
it appears that many states may not be well-situated as they work to meet 
the schedule for implementing new requirements that build upon the 1994 
requirements. 



States successful in meeting key Title I requirements attributed their 
success primarily to four factors. These factors were (1) the efforts of 
state leaders to make Title I compliance a priority; (2) coordination 
between staff of different agencies and levels of government; (3) obtaining 
buy-in from local administrators, educators, and parents; and (4) the 
availability of state-level expertise. Survey respondents identified 
inadequate funding as an obstacle to compliance. The state Title I officials 
we interviewed said that their states’ commitment of resources to norm- 
referenced assessments that conflicted with the 1994 Title I requirements 
contributed to this obstacle. 

Almost 80 percent of the respondents identified state leaders’ efforts as a 
factor that facilitated their meeting the 1994 Title I requirements. In every 
state that had attained compliance with the Title I requirements, the 
officials that we interviewed said that the governor, legislators, or business 
leaders made compliance with the Title I requirements a high-priority. 
States described the development of high-level committees, new state 
legislation, and other measures to raise the visibility and priority of this 
issue. For example, one governor spearheaded a plan that used 
commissions to develop content standards and assessments aligned with 
those standards. Some state officials we interviewed reported that efforts 
by state department of education leaders resulted in msyor organizational 
changes in the state education department. For example, according to one 
Title I Director, the state changed the organizational structure and 
reporting relationships of state offices to organize them by function rather 
than by funding streams and to enhance coordination; according to 
smother Title I Director, state leaders who did not support changes 
necessary to achieve compliance with Title I were replaced with staff that 
did support the changes. 

In responding to our survey, over 80 percent of the Title I officials 
identified the ability of staff or agencies to coordinate their efforts with 
one smother as a factor that helped them meet requirements. In our 



States Cited Support 
and Coordination as 
Furthering 
Compliance 
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interviews^ state officials cited the necessity of-coordination-between state- 

and local staff working in the areas of assessment, instruction, and 
procurement. Two of the states we interviewed specifically noted that 
when the assessment office shared a physical location with the Title I 
office, coordination was easier and the ability to achieve compliance with 
Title I was enhanced. 

Title I and other officials we interviewed in those states that had met the 
1994 Title I assessment requirements noted that they had made great 
efforts to obtain buy-in from other state officials, local administrators, 
educators, and the public. They said that efforts to ensure buy-in paved the 
way for changes meant to ensure compliance with the assessment and 
accountability requirements of the 1994 legislation. Several officials we 
interviewed reported holding public meetings and focus groups to obtain 
input from parents, teachers and local administrators regarding how the 
state should implement Title I requirements. They also reported 
conducting public relations campaigns to educate the public about the 
importance of complying with Title I requirements for standards-based 
assessment. One state, for example, conducted 6 years of focus groups and 
hearings and conducts a conference annually to allow local education 
officials to gain advice from experts regarding any concerns or problems 
they are having in implementing Title I requirements. 

In responding to our survey, over 80 percent of state Title I directors 
identified the availability of state level expertise as a factor that facilitated 
their efforts to meet Title I requirements. State officials we interviewed 
reported that training for teachers and district personnel was often needed 
to apply new content standards in the classroom and to administer 
assessments correctly. Two states, for example, used regional centers to 
educate local staff on assessments and standards. 

Fifty percent of survey respondents identified inadequate funding as an 
obstacle in moving toward compliance and noncompliant states cited this 
problem more often than compliant ones. In our interviews, Title I and 
assessment officials from noncompliant states reported that progress 
toward compliance with Title I requirements was stalled because of 
investments they had made in assessment systems that predated and 
conflicted with the requirements of the 1994 Title I reauthorization. 
Respondents said that they had made substantial investments of time and 
money in systems of assessment that often relied upon norm-referenced 
assessments and did not meet the 1994 requirement for criterion-based 
tests. They noted that it took their states several years to change from the 
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old system of assessment to one meeting the requirements specified by the 
1994 reauthorizing legislation. According to the officials we interviewed, 
building support to start again on another system and obtaining the 
funding made it more difficult to make the necessary changes in a timely 
manner. In addition, one survey respondent from a very small state noted 
that due to the state’s size it has a small number of staff and does not have 
the technical expertise needed to develop a new system, thus hampering 
the state’s ability to meet the requirements. 


Most States are 
Taking Some Action 
To Ensure Accurate 
Scoring, Justification 
for Exemptions, and 
Appropriate 
Accommodations, but 
Actions are Limited 


Most states are taking some action to ensure that Title I assessments are 
scored accurately, that any exemptions for students with limited English 
proficiency are justified, and that students are receiving appropriate 
accommodations when these are needed to gather an accurate assessment 
of their abilities. Most states hire a contractor to score Title I assessments 
and about two-thirds of these states monitored the scoring performed by 
the contractor. Some states that hire contractors have found errors in the 
scoring the contractors did, and in some cases, these errors have had 
serious negative consequences for schools and students. Most states 
reported taking some actions to ensure that students with limited English 
proficiency and disabilities received appropriate accommodations during 
testing. Education is redesigning its current compliance and monitoring 
program to better monitor states’ implementation of Title I. 


Problems Found in 
Contractor Scoring 


According to our survey results, most states (44) hire a contractor for test 
scoring, but 16 of these states identified no monitoring mechanism to 
ensure the accuracy of their contractor’s scoring and reporting. Among 
those states that did report one or more monitoring mechanisms, 15 
reported that they monitored the contractor’s scoring by comparing a 
sample of original student test results to the contractor’s results. A few 
states also reported, in interviews with us, that they compared their most 
recent test scores with those from previous years and looked for 
significant variations that suggested potential errors in scoring. However, 
in our interviews, some assessment officials indicated that they use this 
type of monitoring rather informally. 

The problems identified in assessment scoring suggest that these 
approaches do not always provide adequate assurance of complete and 
accurate results. Indeed, several of the states that use contractors to score 
tests reported that they have had problems with errors in scoring whether 
or not they had monitoring measures in place. In some cases, contractors 
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marked correct answers as incorrect and in other cases the contractors 

calculated the scores incorrectly. The errors were discovered by a number 
of individuals, including local district officials, parents, and state agency 
staff. These scoring errors had impacts on students, families, and school 
and district resources. Based on erroneous scores calculated by a 
contractor, one state sent thousands of children to summer school in the 
mistaken belief that their performance was poor enough to meet the 
criterion for summer intervention. In addition to disrupting families’ 
summer plans and potentially preventing student promotions, this may 
have drawn resources away from other necessary activities. In another 
case, based on a contractor’s erroneous scoring, a state incorrectly 
identified several schools as “in need of improvement,” a designation that 
carries with it both bad publicity and extra expense, for example, districts 
may have to fund the needed improvements. 

A few state officials that we interviewed told us that they have begun 
instituting processes to check the accuracy of scoring. For example, three 
states said that they had hired individuals who were experts in test scoring 
or they hired other third parties to conduct independent audits. States that 
were in compliance with 1994 Title I assessment requirements generally 
had more complete monitoring systems, including measures such as 
technical advisory committees to review results, conduct site visits, and 
follow a sample of tests through the scoring and reporting process. In 
contrast, several states indicated they are still relying on contractor self- 
monitoring to ensure accurate scoring. 

Although Education is obligated under the Federal Managers’ Financial 
Integrity Act of 1982 s and the Single Audit Act 6 to ensure that states that 
receive federal funds comply with statutory and regulatory requirements 
to monitor contractors, it currently takes limited action regarding states’ 
monitoring of assessment contractors. Education’s inspector general has 
reported deficiencies in an important vehicle for such oversight - - 
Education’s compliance reviews of state programs. 7 The compliance 
reviews are conducted on a 4-year cycle and include an on-site visit that 



6 U.S. General Accounting Office, “Standards for Internal Control in the Federal 
Government” GAO/AIMD-OO-21.3.1, (Washington, D.C.: Nov. 1999). 

6 P.L. 98-502. 

7 See “Review of the Office of Elementary and Secondary Education’s Monitoring of 
Formula Grants: Final Audit Report” (ED-OIG/A04-A0013), November 2001. 
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lasts 1 week. Specifically, the OIG cited insufficient time to conduct the 
reviews, lack of knowledge among Education staff about areas they were 
reviewing, and a lack of consistency in how the reviews were conducted. 
Senior Education officials told us the department is redesigning the 
current compliance and monitoring program used for its on-site visits to 
better focus on outcomes and accountability in Title I and that it is 
addressing the OIG’s recommendations. However, a senior Education 
official who is working on the redesign of the compliance reviews told us 
that the current draft plans did not include specific checks on state 
monitoring of assessment scoring. Confidence in the accuracy of test 
scoring is critical to acceptance of the test results’ use in assessing school 
performance. 



Some Actions Taken To 
Ensure Exemptions 
Justified and 
Accommodations 
Appropriate 



According to our surveys and interviews, 33 states have taken at least 
minimal actions to ensure any exemptions for students with limited 
English proficiency are justified and 41 states take actions to ensure 
accommodations for students with disabilities are appropriate. 8 Most 
states reported that they had developed standards for districts to follow in 
accommodating these students so that assessments could yield accurate 
measures of their performance. However, states reported few actions that 
would ensure that these guidelines were being followed. For example, 17 
states reported that they compare the number of students with limited 
English proficiency tested within a given year against the number for the 
previous year. They used this comparison as their means of verifying that 
the numbers of students receiving exemptions were reasonable. As the 
pool of students in a particular school can change substantially from year 
to year, this comparison has obvious limitations. Moreover, students’ 
status, for example with respect to English proficiency, can change from 
year to year. Similarly, 37 states reported using an annual comparison of 
the number of students with disabilities being tested as a check for 
appropriate accommodations. However, it is not evident how such 
comparisons would allow states to ascertain the appropriateness of the 
accommodations. 

Survey results and interviews did indicate that more states are taking 
actions to monitor accommodations for students with disabilities than for 



8 Accommodations for testing modify the circumstances for the student while taking a test 
For example, students with learning disabilities may be allowed more time to take a test 
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students-with-limited English-proficiency7-For example, while 25 states 

reported that they had standards for accommodating students with limited 
English proficiency, 36 had standards for accommodating students with 
disabilities. The state officials that we interviewed told us that this was 
because districts built upon steps they had taken under the Individuals 
with Disabilities Education Act (IDEA) 5 to document the accommodations 
needed by students with disabilities. In general, states said that the 
districts have more experience and technical expertise for assessing and 
supporting students with disabilities because of working under IDEA for 
many years. In contrast, some states lacked consistent standards for 
identifying students with limited English proficiency and more states were 
still working to develop alternate assessments or accommodations for 
these students. Augmenting the 1994 requirements, the new 2001 
legislation requires that states annually assess the language proficiency of 
students with limited English proficiency by the 2002-03 school year. 

States do conduct cyclical monitoring of the implementation of all their 
programs that might be used to assess the appropriateness of district 
policy and practice with regard to testing accommodations. However, in a 
recent review, 10 we found that states varied dramatically in the frequency 
of their on-site visits. The average time between visits to districts ranged 
from 2 years or less (6 states) to more than 7 years (17 states). 



Conclusions This snapshot of the states’ status with respect to the 1994 Title I 

requirements suggests that many states may not be well-positioned to 
meet the requirements added in 2001. Only 17 states were in compliance 
with the assessment requirements of the 1994 law in March of 2002; 
therefore, the majority of states will still be working on meeting the 1994 
requirements as they begin work toward meeting the new requirements. 

In addition, despite the enhanced emphasis on assessment results, states 
still appear to be struggling with ensuring that assessment data are 
complete and correct. The 1994 and 2001 ESEA reauthorizations raised 
student assessments to a new level of importance. The assessments are 



9 IDEA was enacted in 1970 and, among other things, requires schools to provide 
assessments for students with disabilities and provide whatever accommodations they may 
need, including specific accommodation for testing, such as more time to take tests, to 
ensure that the tests provide valid reflections of their abilities. 

10 GAO/HEHS-OO-89, June 1, 2000. 
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intended to help ensure that all students, including those who have 
disabilities and those who have limited English proficiency are meeting 
challenging standards. In addition, assessment results are a key part of the 
mechanism for holding both schools and states accountable for improving 
educational performance. Thus, ensuring the completeness and accuracy 
of assessment data is central to measuring students’ progress and ensuring 
accountability. Without adequate oversight of assessment scoring, efforts 
to identify and improve low-performing schools could be hindered by lack 
of confidence in assessment results or uncertainty regarding whether 
particular schools have been appropriately identified for improvement. 
Education’s current monitoring does not include specific oversight of how 
states ensure the quality of scoring contractors’ work, but Education’s 
revision of its monitoring process provides the agency with an opportunity 
to help states ensure that scoring done by contractors is accurate. 


Recommendation 


To enhance confidence in state assessment results, we recommend that 
when the Department of Education monitors state compliance with 
federal programs, it include checks for contractor monitoring related to 
Title I, Part A. Specifically, Education should include in its new 
compliance reviews a check on the controls states have in place to ensure 
proper test scoring and the effective implementation of these controls by 
states. 


Agency Comments 


We provided Education with a draft of this report for review. The 
Department’s official comments are printed in appendix IE. In its 
comments, Education agreed with our recommendation. Education also 
provided us with technical comments that we incorporated in the report as 
appropriate. 




We are sending copies of this report to appropriate congressional 
committees and other interested parties. If you have any questions about 
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this report, please contact me on (202) 512-7215 or Betty Ward-Zukerman- 
on (202) 512-2732. Key contributors to this report were Mary Roy, G. Paul 
Chapman II, Laura Pan Luo, Corinna Nicolaou, and Patrick DiBattista. 

Sincerely yours, 





Mamie S. Shaul, Director 
Education, Workforce, and 
Income Security Issues 
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Appendix I: Scope and Methodology 



We conducted this review in conjunction with our partners in the 
Domestic Working Group. The Domestic Working Group’s objective is to 
allow officials in the federal, state, and local governmental audit 
communities to interact on a personal and informal basis on various topics 
of mutual concern. The group consists of 18 (6 federal, 6 state, and 6 local) 
top officials and is intended to complement the work of the 
intergovernmental audit forums and other professional associations. 

For this review, the Texas State Auditor's Office conducted a detailed 
assessment of data quality at the state and local levels in Texas, while the 
Department of Education's Office of Inspector General did so at the state 
and local levels in California and conducted additional work on control 
processes at the Department of Education. In Pennsylvania, the 
Pennsylvania Department of Auditor General conducted work at the state 
level and the Philadelphia Controller's Office pursued the same goal within 
the city of Philadelphia. 

To complement these efforts, GAO surveyed all states and conducted 
detailed interviews with several regarding their experiences in 
implementing major provisions of Title I. Specifically, we reviewed three 
key questions: (1) the status of states’ compliance with key 1994 Title I 
assessment requirements; (2) factors that have hindered or helped states 
move toward meeting the requirements; and (3) the actions states are 
taking to ensure that Title I assessments are scored accurately, 
exemptions for students with limited English proficiency are justified, and 
students with disabilities are accommodated during testing according to 
federal regulations. 

We obtained information on the first objective from the Department of 
Education. We met with Education officials and obtained updated listings 
of compliance throughout the audit. In addition, we reviewed state 
decision letters, peer reviews of state assessment systems, and reports 
completed or commissioned by Education’s Planning and Evaluation 
Service. 

To address the second and third questions, we used both a state survey of 
Title I directors and detailed interviews with state Title I offici als and 
other state officials who played a key role in Title I compliance - often 
assessment officials and sometimes Special Education, program 
evaluation, and information technology officials. We sent the survey to all 
50 state directors and to the District of Columbia and Puerto Rico. We 
received 50 completed surveys. We followed up with 19 states to clarify 
and expand on questions in the interview related to contracting for the 






O/f 



Page 21 



GAO-02-393 Title I Assessments 



Appendix I: Scope and Methodology 



scoring of tests. We interviewed officials from 5 states that-had assessment 

systems approved by Education and 3 states that were still trying to attain 
compliance. We also interviewed two expert reviewers, Education officials 
with responsibility for Title I and program review, three officials at 
Education’s regional assistance centers, and officials at the Council of 
Chief of State School Officers. We coordinated our work and findings with 
our audit partners, who provided us with information relative to their 
states’ activities. 



Appendix II: Comments from the Department 
of Education 




UNITED STATES DEPARTMENT OF EDUCATION 

OFFICE OF ELEMENTARY AND SECONDARY EDUCATION 

THE ASSISTANT SECRETARY 



2 i 



Mamie S. Shaul 
Director 

Education, Workforce and Income Security Issues 
U.S. General Accounting Office 
Washington, DC 20548 

Dear Ms. Shaul; 

This is in response to your letter to Secretary Paige requesting Department of Education 
comments on the draft report Title /: Education Needs to Monitor States ' Scoring of 
Assessments (GAO/02-393). Thank you for the opportunity to comment. 

The draft document recommends that the Department monitor States* actions with 
respect to their contractors’ scoring of assessments. As a matter of clarification, scoring 
of assessments is not typically a State function. Often, the State enters into a contract 
with a test developer who, as a part of the contractual agreement, is responsible for 
scoring. States have the primary responsibility to oversee the work of the test developer 
and to verify the accuracy of results. Typically, States include scoring in their contracts 
with test developers, as well as other safeguards related to administration, analysis, and 
reporting procedures. 

The Department, as recommended, will include in its monitoring activities a review of a 
State’s compliance in monitoring the technical quality of products delivered by its 
contractor, and its procedures to ensure accuracy of assessment data prior to 
dissemination. 



If I can be of further assistance, please do not hesitate to contact me at (202) 401-01 13. 



Sincerely, » 

Mi&U k UyU_ 



Susan B. Neuman, Ed.D. 



600 INDEPENDENCE AVE . S.W. WASHINGTON. D.C. 20202-6100 
Our mission Is to ensure equal access to education and to promote educational excellence throughout the Nation. 
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GAO’s Mission 


The General Accounting Office, the investigative arm of Congress, exists to 
support Congress in meeting its constitutional responsibilities and to help 
improve the performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates federal 
programs and policies; and provides analyses, recommendations, and other 
assistance to help Congress make informed oversight, policy, and funding 
decisions. GAO’s commitment to good government is reflected in its core values 
of accountability, integrity, and reliability. 


Obtaining Copies of 
GAO Reports and 
Testimony 


The fastest and easiest way to obtain copies of GAO documents is through the 
Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of 
current reports and testimony and an expanding archive of older products. The 
Web site features a search engine to help you locate documents using key words 
and phrases. You can print these documents in their entirety, including charts and 
other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 
daily. The list contains links to the full-text document files. To have GAO e-mail 
this list to you every afternoon, go to www.gao.gov and select "Subscribe to daily 
e-mail alert for newly released products" under the GAO Reports heading. 


Order by Mail or Phone 


The first copy of each printed report is free. Additional copies are $2 each. A 
check or money order should be made out to the Superintendent of Documents. 
GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a 
single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 
P.O. Box 37050 
Washington, D.C. 20013 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 


Visit GAO’s Document 
Distribution Center 
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Room 1100, 700 4th Street, NW (comer of 4th and G Streets, NW) 
Washington, D.C. 20013 


To Report Fraud, 
Waste, and Abuse in 
Federal Programs 


Contact: 

Web site: www.gao.gov/fraudnet7fraudnet.htm, 

E-mail: fraudnet@gao.gov, or 

1-800-424-5454 or (202) 512-7470 (automated answering system). 


Public Affairs 
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